To RCT or not to RCT? The ongoing saga of randomised trials in quality improvement.

To cite: Parry G, Power M. BMJ Qual Saf Published Online First: [please include Day Month Year] doi:10.1136/bmjqs-2015004862 Williams et al describe a well-conducted cluster randomised trial of a stoke quality improvement (QI) initiative, which aimed to improve two inpatient stroke indicators with strong evidence linking them to improved patient outcomes. They randomised five hospitals to receive a QI intervention, and six to receive only indicator feedback. In aggregate, they found evidence of improvement in one indicator, in the intervention group, relative to the control, but this was not sustained once the intervention period ended. The design, execution and analysis of the study were textbook for a cluster randomised controlled trial (RCT) design, aligning well with the CONSORT statement, the gold standard for RCT execution. There is much debate within the improvement field about the value of RCTs in determining the effectiveness of improvement interventions. In 2007, Donald Berwick’s monologue ‘eating soup with a fork’ provided a convincing argument for why the RCTwas necessary for evidence-based medicine, but inadequate for evaluating complex social interventions such as collaboratives and campaigns. Since then, there has been an apparent ‘cooling’ in the appetite of improvement practitioners to adopt RCT methods in attempts to understand the overall impact of improvement initiatives. Against this backdrop, we applaud the authors in their attempt, which goes against the trend, but disappointingly, once again, offers conflicting and weak evidence of beneficial effect despite adherence to rigorous method. So what does this study teach us about whether or not to embrace RCTs in improvement? It is becoming widely recognised that improvement is a complex function of the what ––the changes that are being sought, the how—the method, rationale or theory for the wider improvement initiative, the motivation and capability of the organisations (both the improvement teams and the delivery team) involved and the context within which the improvement work is occurring. Moreover, it is clear from the literature that the policy context, the financial climate, the attention of regulators and professional bodies, the voice of patients and the fabric of the organisational ecosystem are critical to the likelihood of success. Improvement designers are starting to use tools such as the Model for Understanding Success in Quality (MUSIQ) to understand context. In the study by Williams et al, it is unclear how much ‘up front’ work was done to understand, influence or modify the context. Indeed, this could be argued as one of the blind spots of the RCT. In addition to the ecosystem, the behaviours and motivations of teams entering improvement collaboratives appear to be as important as the environmental context. For example, in Stroke 90:10 negative and positive behaviours from teams were present in equal measure with observations of social loafing and freeriding. 5 These behaviours go some way to explaining the likelihood of a team to ‘benefit’ from the improvement intervention as designed and currently are best investigated through qualitative research methods such as ethnography and semistructured interview. We know from previous studies that the nature or the what of an improvement intervention is often complex, adapts over time and can be poorly described, leading to inapt designs and ambiguous results. In the study by Williams et al, the what is simple and well described; yet, the performance on two of the primary outcomes suggest that deep vein thrombosis (DVT) prophylaxis and the implementation of EDITORIAL